Implementation of custom bytetrack demo for real time stream analysis [GSoC'2026] by Vishwa2684 · Pull Request #4420 · openvinotoolkit/model_server

Vishwa2684 · 2026-07-30T13:23:09Z

Summary

This PR adds a ByteTrack Real-Time Stream Analysis demo for OpenVINO Model Server (OVMS).

The demo reuses the MediaPipe ByteTrack graph from demos/mediapipe/bytetrack with the yolox_tiny model and integrates it with the real_time_stream_analysis Python client. This enables object detection and multi-object tracking on RTSP streams using OVMS.

Key Changes

Moved the ByteTrack implementation from the MediaPipe OVMS fork into the Model Server repository:
Implementation of custom bytetrack graph [GSoC'26] mediapipe#131
Added a custom OVYoloXTensorsToDetectionsCalculator to process YOLOX output tensors from the YOLOX-Tiny model hosted on Hugging Face.

Directory Structure

Directory	Description
`demos/mediapipe/bytetrack`	MediaPipe ByteTrack OVMS demo and configuration
`src/bytetrack`	ByteTrack utilities and ByteTrack calculator
`src/yolox`	Utilities for processing YOLOX-Tiny output tensors and converting them into detections

What's Included

A new MediaPipe ByteTrack demo in demos/mediapipe/bytetrack.
Setup instructions covering model conversion, OVMS deployment, and RTSP streaming.
A client.py example for running inference with:
- Video input and output.
- RTSP input and output streams.

Copilot

Pull request overview

This PR adds a new MediaPipe ByteTrack demo that integrates OVMS + the existing real-time stream analysis Python client to run object detection + multi-object tracking on RTSP streams. It also updates the MediaPipe build integration and adjusts streaming timestamp handling in the KFS graph executor.

Changes:

Adds a new demos/mediapipe/bytetrack demo (graph, OVMS config, and setup README).
Extends the real-time stream analysis Python demo CLI to pass FFmpeg output dimensions through to StreamClient.
Updates MediaPipe/Bazel integration (new ByteTrack calculators dependency; changes MediaPipe git_repository source) and adds stricter timestamp monotonicity checks for streaming.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`WORKSPACE`	Switches the MediaPipe repository source (currently to a fork/branch).
`third_party/mediapipe_calculators/BUILD`	Adds ByteTrack calculators target to the OVMS MediaPipe calculators aggregation library.
`src/kfs_frontend/kfs_graph_executor_impl.cpp`	Changes timestamp deserialization/defaulting and adds stream-level monotonic timestamp enforcement.
`demos/real_time_stream_analysis/python/client.py`	Adds CLI args for FFmpeg output width/height and forwards them to `StreamClient`.
`demos/mediapipe/bytetrack/README.md`	Introduces demo setup/run instructions for ByteTrack RTSP streaming demo.
`demos/mediapipe/bytetrack/config.json`	Adds OVMS config for the YOLOX model + ByteTrack MediaPipe graph.
`demos/mediapipe/bytetrack/bytetrack_ovms.pbtxt`	Adds the MediaPipe graph wiring OpenVINO/OVMS inference + ByteTrack tracking + overlay.

Suppressed comments (1)

src/kfs_frontend/kfs_graph_executor_impl.cpp:1147

The inline comment "// this method" is not descriptive. Either remove it or replace with a comment that explains what the function does.

// this method
Status createAndPushPacketsImpl(

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

dtrawins · 2026-08-11T15:34:07Z

make sdl-check is failing with error:
No header files detected:
./demos/mediapipe/bytetrack/bytetrack_ovms.pbtxt
Could you add it to exception list in ci/lib_search.py

Vishwa2684 · 2026-08-12T11:28:28Z

@atobiszei @dtrawins I've written code for custom calculator OVYoloXTensorsToDetections in src/yolox, and wrote a script to download OpenVINO/yolox_tiny-fp16-ov model weights and write class labels from the model config.

Vishwa2684 · 2026-08-13T18:23:02Z

@dtrawins Why is the ci failing? Could you share the logs?

…List

Copilot

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 16 comments.

+        return absl::OkStatus();
+    }
+
+    absl::Status Process(CalculatorContext* cc) final {


…server into gsoc_bytetrack

…gnments

atobiszei · 2026-09-02T06:52:50Z

+    float score_ = 0.f;
+    bool is_activated_ = false;
+    TrackState state_ = TrackState::NEW;
+    static std::atomic<int> count_;


This is static - one instance per OVMS instance. OVMS allows for several concurrent inferences at a time - don't we have a bug here? We should have a bytetrack stored per graph/calculator instance not per process.

atobiszei · 2026-09-02T07:05:30Z

+
+namespace mediapipe {
+
+class DetectionColorByIdCalculator : public CalculatorBase {


Coudl we add some simple test for this calculator using mediapipe::CalculatorRunner and for ov_yolox_tensors_to_detections?

atobiszei · 2026-09-02T07:13:40Z

+};
+
+// IoU between two [top, left, bottom, right] boxes
+inline float ComputeIoU(const Eigen::Vector4f& a,


There's a lot of utilities functions in PR that could end in exceptions. We should defend against such cases at some point, as hackers could try to do DoS on OVMS with malicious inputs.

Vishwa2684 and others added 7 commits July 14, 2026 15:15

Use Vishwa2684/mediapipe_ovms for bytetrack demo

58ae1cc

Add ffmpeg_output width and height as client params for rtsp sink

ed27dc3

Added bytetrack demo which can be used with RTSP

1ad498f

Merge branch 'openvinotoolkit:main' into gsoc_bytetrack

dc02a3a

Using input_video instead of transformed_input_video for overlay

4ec3459

Update docs

89e096e

add logic to check timestamps

f1615da

dtrawins requested a lite review from Copilot August 11, 2026 13:46

Copilot started reviewing on behalf of dtrawins August 11, 2026 13:47 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

dtrawins reviewed Aug 11, 2026

View reviewed changes

Comment thread demos/mediapipe/bytetrack/README.md Outdated

formatting in client.py

155f27e

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Vishwa2684 requested a review from dtrawins August 12, 2026 11:28

Write a custom calculator to process yolox tiny's ov tensors

fdb2dac

Vishwa2684 force-pushed the gsoc_bytetrack branch from 0a2781e to fdb2dac Compare August 12, 2026 14:51

Move bytetrack logic from seperate mediapipe repo to OVMS

a7a25d9

Vishwa2684 force-pushed the gsoc_bytetrack branch 2 times, most recently from 4a0b8f8 to bf1c022 Compare August 13, 2026 18:13

Add license headers to files

7524ee6

Vishwa2684 force-pushed the gsoc_bytetrack branch from bf1c022 to 7524ee6 Compare August 13, 2026 18:19

Vishwa2684 force-pushed the gsoc_bytetrack branch 7 times, most recently from 7d36dd6 to 2b2ba29 Compare August 19, 2026 22:07

Vishwa2684 force-pushed the gsoc_bytetrack branch from b5da86c to 0369140 Compare August 21, 2026 15:29

Vishwa2684 and others added 2 commits August 24, 2026 18:50

Add ByteTrack calculators to WhitelistRegistered.MediapipeCalculators…

0610240

…List

Merge branch 'main' into gsoc_bytetrack

cc871ad

atobiszei requested a balanced review from Copilot August 26, 2026 12:33

Copilot started reviewing on behalf of atobiszei August 26, 2026 12:34 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Vishwa2684 and others added 6 commits August 26, 2026 19:14

Merge branch 'main' into gsoc_bytetrack

6e96f00

add os.makedirs to create directory

d8f82c5

Merge branch 'gsoc_bytetrack' of https://github.com/Vishwa2684/model_…

8edaf6d

…server into gsoc_bytetrack

Minor change in README

edde13f

Remove removed_stracks_ in ByteTrackCalculator

c3eb6af

use atomic variable for counter to prevent data race

76e1bbf

Vishwa2684 force-pushed the gsoc_bytetrack branch from 3f64593 to 76e1bbf Compare August 27, 2026 17:28

Refactor LinearAssignment to improve condition checks for column assi…

d413dd6

…gnments

atobiszei reviewed Aug 31, 2026

View reviewed changes

Remove dead code and relocate debug calculator

1e7eb39

Vishwa2684 requested a review from atobiszei August 31, 2026 17:00